Field of the Invention
The invention relates to a method and a system for inside-out optical-inertial tracking a movable object and it may be used in virtual reality or augmented reality systems, internal logistics systems (including factory logistics, warehouse logistics or store logistics), robotics systems, unmanned movable objects and other types of industrial, scientific or training systems.
Description of the Related Art
Tracking a movable object in a virtual reality (VR) system or in an augmented reality (AR) system means tracking position and orientation of a movable object, which are used for rendering an image displayed to a user of the system.
A tracking system may be configured either as an outside-in tracking system, where a sensor or multiple sensors is (are) located in an area outside a tracked object and a reference point or multiple reference points is (are) located on the tracked object, or an inside-out tracking system, where a sensor or multiple sensors is (are) located on a tracked object and a reference point or multiple reference points is (are) located in an area outside the tracked object.
As for sensor types, there is optical tracking, where optical sensors are used, e.g., cameras operating in visible light or infrared light range, and inertial tracking, where inertial sensors are used, e.g., gyroscopes and accelerometers. Moreover, magnetic field sensors (magnetometers), sensors of height above the sea level (altimeters) and some other sensor types may also be used in tracking systems.
There are tracking systems using centralized positioning systems, which may be local or global, satellite-based or terrestrial ones.
Most of tracking systems intended for use in gaming, entertainment and training VR/AR systems are combined tracking systems. In particular, optical-inertial tracking systems are widely used, which are based on combined application of optical and inertial sensors, where the sensors complement each other and mutually compensate for each other's drawbacks.
There are tracking systems implementing an inside-out optical-inertial tracking approach, where a video camera is located on a movable object and optical light-emitting or retroreflective markers used as reference points for optical tracking are fixedly installed on a site or in a territory of the system operational area in a predefined manner A gyroscope and an accelerometer are also located on the movable object and data acquired form these sensors is used for determination of position, orientation, velocity and direction of movement, acceleration and other parameters of the tracked object, along with data acquired form the video camera. Data extrapolation is employed for prediction of the above-indicated parameters, so as to compensate algorithm-dependent and hardware-dependent delays.
U.S. Pat. No. 9,352,230 discloses an outside-in optical-inertial tracking system, where camera, gyroscope and accelerometer data is used and data extrapolation is employed for prediction of parameters of a movable object. A drawback of this system is the need for application of multiple fixed cameras for covering the whole operational area providing enough room for gaming, entertainment and training VR/AR systems. A possibility of implementation of an inside-out optical-inertial tracking approach is mentioned in the document; however, technical details of such an implementation are not disclosed.
US Patent Publication No. 2017177937 discloses an inside-out optical-inertial tracking system, where camera, gyroscope and accelerometer data is used and data extrapolation is employed for prediction of parameters of a movable object. This system does not employ markers installed on a site or in a territory of the system operational area in a predefined manner and used as reference points for optical tracking. Using feature points of architectural or natural environment as reference points increases requirements for computational capability of the system and when the computational resources are limited, this approach inevitably causes a decrease in the system operation speed.
US Patent Publication No. 2008300055 discloses an outside-in optical-inertial tracking system, where camera, gyroscope and accelerometer data is used and data extrapolation is employed for prediction of parameters of a movable object. A drawback of this system is the need for application of multiple fixed cameras for covering the whole operational area and providing enough room for gaming, entertainment and training VR/AR systems. A possibility of implementation of an inside-out optical-inertial tracking approach is mentioned in the document, wherein, where the tracking camera is located in a bangle; however, technical details of this implementation option are not disclosed.
US Patent Publication No. 2016232715 discloses an outside-in optical-inertial tracking system, where camera, gyroscope and accelerometer data is used and data extrapolation is employed for prediction of parameters of a movable object. A drawback of this system is a need for application of multiple fixed cameras for covering the whole operational area providing enough room for gaming, entertainment and training VR/AR systems. A possibility of implementation of an inside-out optical-inertial tracking approach is mentioned in the document, where the tracking camera is located in a mobile device; however, technical details of this implementation option are not disclosed.
US Patent Publication No. 2012105473 discloses an outside-in optical-inertial tracking system, where data acquired from camera, gyroscope and accelerometer located in a helmet display is used and data extrapolation is employed for prediction of parameters of a movable object. This system does not employ active markers; this feature increases requirements for computational capability of the system and when the computational resources are limited, this approach inevitably causes a decrease in the system operation speed.
US Patent Publication No. 2015029218 discloses an outside-in optical-inertial tracking system, where data acquired from camera, gyroscope and accelerometer located in a helmet display is used and data extrapolation is employed for prediction of parameters of a movable object. This system does not employ active markers, thus requirements for computational capability of the system are increased and when the computational resources are limited, this approach inevitably causes a decrease in the system operation speed.
Patent Publication WO2014210342 discloses an outside-in optical-inertial tracking system, where data acquired from camera, gyroscope and accelerometer located in a helmet-mounted display is used and data extrapolation is employed for prediction of parameters of a movable object. This system does not employ active markers; this feature increases requirements for computational capability of the system and when the computational resources are limited, this approach inevitably causes decrease in the system operation speed.
US Patent Publication No. 2014354515 discloses an outside-in optical-inertial tracking system, where camera, gyroscope and accelerometer data is used and data extrapolation is employed for prediction of parameters of a movable object. A drawback of this system is necessity of application of multiple fixed cameras for covering the whole operational area providing enough room for gaming, entertainment and training VR/AR systems. A possibility of implementation of an inside-out optical-inertial tracking approach is mentioned in the document, where the tracking camera is located in a helmet-mounted display; however, technical details of such an implementation are not disclosed.
US Patent Publication No. 2017185171 discloses an outside-in optical-inertial tracking system, where camera, gyroscope and accelerometer data is used and data extrapolation is employed for prediction of parameters of a movable object. A drawback of this system is necessity of application of multiple fixed cameras for covering the whole operational area providing enough room for gaming, entertainment and training VR/AR systems. A possibility of implementation of an inside-out optical-inertial tracking approach is mentioned in the document, where the tracking camera is located in close proximity to a user; however, technical details of this implementation option are not disclosed.
US Patent Publication No. 2017018121 discloses an outside-in optical-inertial tracking system, where camera, gyroscope and accelerometer data is used and data extrapolation is employed for prediction of parameters of a movable object. A drawback of this system is a need for application of multiple fixed cameras for covering the whole operational area providing enough room for gaming, entertainment and training VR/AR systems. A possibility of implementation of an inside-out optical-inertial tracking approach is mentioned in the document, where the tracking camera is located in close proximity to a user; however, technical details of this implementation option are not disclosed.
US Patent Publication No. 2014062881 discloses an outside-in optical-inertial tracking system, where data acquired from camera, gyroscope and accelerometer located in a helmet-mounted display is used and data extrapolation is employed for prediction of parameters of a movable object. This system does not employ active markers, thus requirements for computational capability of the system are increased and when the computational resources are limited, this approach inevitably causes decrease in the system operation speed.
Non-patent document Hogue et al. [1] discloses an inside-out optical-inertial tracking system, where data acquired from camera, gyroscope and accelerometer located in a helmet-mounted display is used and data extrapolation is employed for prediction of parameters of a movable object. Images formed on walls by a laser beam projection are used as reference points for optical tracking instead of active markers; this feature increases requirements for computational capability of the system and when the computational resources are limited, this approach inevitably causes decrease in the system operation speed.
Non-patent document Wormell et al. [2] discloses an inside-out optical-inertial tracking system, where data acquired from camera, gyroscope and accelerometer located in a helmet-mounted display is used and data extrapolation is employed for prediction of parameters of a movable object. Images applied to walls are used as reference points for optical tracking, instead of active markers; this feature increases requirements for computational capability of the system and when the computational resources are limited, this approach inevitably causes decrease in the system operation speed.
Non-patent documents Atac et al. [3, 4] disclose an inside-out optical-inertial tracking system, where data acquired from camera, gyroscope and accelerometer located in a helmet-mounted display is used and data extrapolation is employed for prediction of parameters of a movable object. Barcodes applied to walls are used as reference points for optical tracking instead of active markers; this feature increases requirements for computational capability of the system, and when the computational resources are limited, this approach inevitably causes a decrease in the system operation speed.
A combination of inside-out optical tracking and inertial tracking for implementation of a mobile AR system is discussed in a non-patent document Pinz et al. [5]; however, this discussion is mostly theoretical.
Using optical sensor data for correction of inertial sensor data in an optical-inertial tracking system is discussed in a non-patent document You et al. [6]; however, this discussion is mostly theoretical.
Non-patent document Miezal et al. [7] discloses an inside-out optical-inertial tracking system for tracing position of the user's head, using a single passive marker located in a visibility range of a camera; such an approach limits application of this solution to those options, where the user is relatively sedentary, so its implementation in gaming, entertainment and training VR/AR systems is not expedient.
Therefore, there is a need for a fast, reliable, precise and relatively cheap method of tracking a user in VR/AR systems, where the user's limbs may jerk and the user's body may move fast in a rather wide operational area, whose size may range from several square meters (in single-user systems) to several tens of square meters or even to several hundreds of square meters (in multi-user systems).
Similar tasks related to tracking a movable object are relevant for other fields, in particular, for internal logistics systems (including factory logistics, warehouse logistics or store logistics), robotics systems, unmanned vehicles and other types of industrial, scientific or training systems.